Repository navigation
Conversation
fd94d18 to
0421b3f
Compare
0421b3f to
54c244d
Compare
54c244d to
5cc9908
Compare
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## add-priority-crawler #608 +/- ##
========================================================
+ Coverage 76.44% 76.69% +0.24%
========================================================
Files 86 86
Lines 9927 9949 +22
Branches 1088 1094 +6
========================================================
+ Hits 7589 7630 +41
+ Misses 2129 2106 -23
- Partials 209 213 +4
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
5cc9908 to
c46ded7
Compare
c46ded7 to
4782f44
Compare
4782f44 to
0839be5
Compare
0839be5 to
e76ba1a
Compare
rajeee
left a comment
There was a problem hiding this comment.
Reviewer guide: one note per changed source file, saying what changed and why the existing structure couldn't do it. Tests and docs are not annotated.
There was a problem hiding this comment.
New: the extraction schema for data center ordinances.
A one-shot schema: the model returns a sparse table with one row per feature the ordinance addresses, plus extraction_notes saying what was and was not extracted and why.
Points worth reviewing:
- 35 features, with IDs that mirror the vocabulary of the hand-coded research workbook so output can be compared against it directly. Each has a definition under
$definitionsstating what counts, what value to record, and what to ignore. data_center_typeis a fixed list: Micro, Small, Medium, Large or Hyperscale Data Center, Accessory Data Center, or null for data centers of any size. The extraction call uses strict structured output, so the model cannot emit anything else. This replaced free-text class names, which produced near-duplicates across runs ('Large-Scale Data Centers' / 'Large-Scale Data Center'). The jurisdiction's own class name goes at the start ofsummary.- One row per feature and type. Rows are typed by how the ordinance states each requirement (per tier, or generally). Other distinctions, such as whether a site is part of a unified development plan, are conditions: the most restrictive value is recorded and the others are listed in
summary. - Numeric features carry a single number, the most restrictive one, chosen by the rules in
$instructions.numeric_prioritization. - Drafts and other jurisdictions' text are excluded. Meeting packets often attach a draft ordinance or reproduce neighbouring counties' rules for comparison. Redline markup alone does not make a text a draft.
tests/python/unit/extraction/test_extraction_data_centers_schema.py checks the contract (feature list matches the definitions, the type list, unit and value rules).
There was a problem hiding this comment.
New: plugin configuration for data centers.
Search query templates, website and heuristic keywords, and the extraction system prompt. The prompt names the jurisdiction ({jur_full_name}), which the model was previously never told, and repeats the schema's key rules in prose: the fixed type list, one row per feature and type, most restrictive value, and no rows from drafts or from other jurisdictions' regulations.
No Python is needed for the plugin; __init__.py builds it from this file with the existing create_schema_based_one_shot_extraction_plugin.
There was a problem hiding this comment.
Registers the plugin.
Builds COMPASSDataCentersExtractor from plugin_config.yaml with the existing one-shot factory, under tech="data_centers". The two one-line imports in compass/extraction/__init__.py and compass/__init__.py make it load with the package, the same way the other bundled plugins do.
There was a problem hiding this comment.
Jurisdictions can be selected by FIPS code, and codes keep their leading zeros.
- A jurisdiction CSV with a
FIPScolumn is now matched on that column alone (_load_jurisdictions_by_fips). A blank or unknown code is skipped with the usual warning, without falling back to names. fips_to_strreturns a string unchanged instead of round-tripping it through an integer.
Why: names are ambiguous for cities and townships (the same place name exists in several counties and states), and the data center reference set is keyed by FIPS. Converting through an integer turned 0446000 into 446000, which no longer matches the catalog.
This affects every technology, not only data centers: jurisdiction codes in outputs now keep their leading zero (08001 rather than 8001). The integration test test_collection_cli_selectively_resets_and_collects_missing_shards was updated for that.
There was a problem hiding this comment.
Known documents are looked up by the full code, with a fallback for existing configs.
known_local_docs and known_doc_urls are keyed by jurisdiction code. Since codes now keep leading zeros, _known_source_for_code first tries the full code and then the same code without leading zeros, so a config written with 8001 still finds its documents for 08001.
There was a problem hiding this comment.
Two changes to the shared one-shot components. Both apply to every one-shot plugin, not only data centers.
- The relevance prompt gains a sentence: the content must be about the specific technology or use the schema targets, and a definition or provision for some other use that merely mentions a related activity does not count. Without it, zoning-code definitions chapters were kept as relevant because one definition (for example, a business office that "includes data processing") matched.
- When the parser extracts nothing, it now logs at INFO and includes the schema's
extraction_notes, so the log says why. Schemas without that field log "no explanation given".
There was a problem hiding this comment.
"definition" is added to EXCLUDE_FROM_ORD_DOC_CHECK.
That set lists features that do not, on their own, make a document count as an ordinance. A definitions chapter defines terms but regulates nothing, so a document yielding only definition rows is not accepted as the jurisdiction's ordinance and the pipeline moves on to the next document.
Worth knowing: together with the district features already in this set, this means a document whose only rows are definitions and district lists is discarded. In a recent 22-jurisdiction run that dropped two jurisdictions whose adopted ordinance contains little else.
5cb2385 to
3ec49dd
Compare
3ec49dd to
84e7d7f
Compare
84e7d7f to
a4d65e2
Compare
a4d65e2 to
3248b16
Compare
Adds the data center schema and plugin config, post-processing for data center type labels, and scripts that export the human-extracted reference ordinances. Also loads jurisdictions by FIPS when the input file has a FIPS column, keeps leading zeros in known-source lookups, and stops definitions-only documents from counting as an ordinance.
… qualitative values
3248b16 to
e6bd7ed
Compare
Adds the data center schema and plugin config, post-processing for
data center type labels, and scripts that export the human-extracted
reference ordinances.
Also loads jurisdictions by FIPS when the input file has a FIPS
column, keeps leading zeros in known-source lookups, and stops
definitions-only documents from counting as an ordinance.
Stack created with GitHub Stacks CLI • Give Feedback 💬